Skip to content

Update pipeline targets#47

Merged
jemrobinson merged 30 commits intomainfrom
42-update-pipeline-targets
Aug 12, 2025
Merged

Update pipeline targets#47
jemrobinson merged 30 commits intomainfrom
42-update-pipeline-targets

Conversation

@jemrobinson
Copy link
Copy Markdown
Member

@jemrobinson jemrobinson commented Aug 4, 2025 •
edited
Loading

This PR makes the following breaking changes:

  • Batches of inputs now have an additional time dimension of size n_history_steps
  • Batches of outputs now have an additional time dimension of size n_forecast_steps
  • Timestamps of input and output datasets are synchronised so that for a given date D we take [D, D+1, ... D + n_history_steps-1] as input and [D+n_history_steps, D+n_history_steps+1, ..., D+n_history_steps+n_forecast_steps-1] as output.
    • Note that this also allows the same dataset to be used for input and output (on different dates) closing Add SIC as data input #24
  • The forward method of our models now takes a dict as input instead of a list
    • This is a dict[str, Tensor[NTCHW]] (returning a single Tensor[NTCHW] as output) where
      • N is the batch size
      • T is n_history_steps for input and n_forecast_steps for output
      • C is the number of input channels for inputs and the number of output channels (currently 1 as we're only predicting SIC) for outputs
      • H and W are the input and output sizes (e.g. output size is 432x432 for SIC)

We may want to merge #46 before this one (note I'm on leave until Thursday, so happy to wait until then to take this forward).

Closes #24, #39, #42

@jemrobinson
🔧 Moved trainer and optimizer configs into dedicated files
7766acc
@jemrobinson
🔧 Move prediction target settings to a dedicated config file
4b4f1f1
@jemrobinson
✨ Pass n_forecast_steps and n_history_steps to CombinedDataset
b657ae6
@jemrobinson
🦺 Require that all datasets in a CombinedDataset have the same frequency
2d183f3
@jemrobinson
✨ Add get_forecast_steps and get_history_steps to return lists of dat…
be2916f 
…es for a given start date
@jemrobinson
🎨 Return a dictionary of tensors instead of a list
beeec9d
@jemrobinson
🏷️ Separate numpy and torch batches
b1d834f
@jemrobinson
✨ Get all available dates when constructing a CombinedDataset
0166f67
@jemrobinson
➕ Add cached index_from_date method
664d350
@jemrobinson
squash ✨ Get all available dates when constructing a CombinedDataset
3084f86
@jemrobinson
♻️ Refactor Encoder and Decoder to take and return a time dimension
57d242e
@jemrobinson
♻️ Pass NTCHW batches through training
90fcbbd
@jemrobinson jemrobinson requested review from a team, IFenton and louisavz August 7, 2025 08:32
@jemrobinson jemrobinson force-pushed the 42-update-pipeline-targets branch from 1a43fe5 to 38b2202 Compare August 7, 2025 08:59
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Aug 7, 2025 •
edited
Loading

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)		Lines missing
  ice_station_zebra
  types.py 61, 67
  ice_station_zebra/data/anemoi
  zebra_dataset.py
  ice_station_zebra/data/lightning
  combined_dataset.py 21-39, 60, 71, 82-83, 87, 94, 101-104, 110-113
  zebra_data_module.py 35-44, 56-60, 90, 104, 116, 130, 142, 156
  zebra_dataset.py
  ice_station_zebra/evaluation/callbacks
  plotting_callback.py 58-65
  ice_station_zebra/models
  encode_process_decode.py 23-31, 54, 73-87
  zebra_model.py 27-35, 43, 65, 79-82, 96-98, 112-116
  ice_station_zebra/models/decoders
  __init__.py 1-4
  base_decoder.py 1-24
  naive_latent_space_decoder.py 1-27, 43-66
  ice_station_zebra/models/encoders
  __init__.py
  base_encoder.py 19-21
  naive_latent_space_encoder.py 24-38, 52, 64
  ice_station_zebra/models/processors
  null.py 1-27
Project Total  

This report was generated by python-coverage-comment-action

@jemrobinson jemrobinson force-pushed the 42-update-pipeline-targets branch 2 times, most recently from a0ed297 to 319cbe9 Compare August 8, 2025 12:13
IFenton
IFenton requested changes Aug 8, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@IFenton IFenton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed a few things we should change in coworking

Comment thread ice_station_zebra/config/train/predict/osisaf-south.yaml
Comment thread ice_station_zebra/data/lightning/zebra_data_module.py Outdated
Comment thread ice_station_zebra/data/lightning/zebra_dataset.py
Comment thread ice_station_zebra/models/zebra_model.py Outdated
Comment thread ice_station_zebra/models/zebra_model.py
@jemrobinson jemrobinson requested a review from IFenton August 12, 2025 10:20
IFenton
IFenton approved these changes Aug 12, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@IFenton IFenton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jemrobinson jemrobinson merged commit f564ff9 into main Aug 12, 2025
3 checks passed
@jemrobinson jemrobinson deleted the 42-update-pipeline-targets branch August 12, 2025 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@IFenton IFenton IFenton approved these changes

@louisavz louisavz Awaiting requested review from louisavz

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add SIC as data input

2 participants

@jemrobinson @IFenton